Display system, display apparatus, method for controlling display apparatus, and program

ABSTRACT

An HMD includes an image display section that allows a user to view an image and transmits an outside scene, an evaluation section that evaluates the state of the user, and a communication section that communicates with another HMD. A control section transmits visual field data on the basis of images captured with a right camera and a left camera to the other HMD, causes the communication section to transmit notification data on the basis of a result of the evaluation performed by the evaluation section to the other HMD, and causes the image display section to perform display on the basis of data transmitted from the other HMD.

BACKGROUND

1. Technical Field

The present invention relates to a display system, a display apparatus,and a method for controlling the display apparatus, and a program.

2. Related Art

To assist a person who performs work, there has been a known case wherea display apparatus called a head mounted display (HMD) worn around theperson's head is used (see JP-A-2008-113317, for example). In the systemdescribed in JP-A-2008-113317, the HMD that the worker wears includes anelectronic camera, and an image of a work target captured with theelectronic camera of the HMD is displayed on a host PC operated by aperson who gives an instruction. The person who gives an instructionlooks at the image of the work target captured with the electroniccamera and causes the HMD on the worker to display an alarm or any othertype of information.

In the configuration of related art described above, in which the personwho gives an instruction to assist the worker grasps the state of theworker from the image displayed on the PC, it has been desired that theperson who assists the worker convey a large amount of information in amore understandable form.

SUMMARY

An advantage of some aspects of the invention is to enable a person,when the person assists another person using a head mounted display toperform an action, to give a large amount of information on the actionin a more understandable manner.

An aspect of the invention relates to a system including a plurality ofhead-mounted-type display apparatus, and each of the display apparatusincludes a display section that allows a user to view an image andtransmits an outside scene, a communication section that communicateswith another display apparatus of the plurality of display apparatus, animaging section that performs image capturing over a range that overlapswith a visual field of the user, an evaluation section that evaluates astate of the user, and a control section, and the control sectiontransmits visual field data on the basis of an image captured by theimaging section to the another display apparatus, causes thecommunication section to transmit notification data on the basis of aresult of the evaluation performed by the evaluation section to theanother display apparatus, and causes the display section to performdisplay on the basis of data transmitted from the another displayapparatus.

According to the aspect of the invention, in which data on the visualfields and data on inputs can be shared among the plurality of displayapparatus, a plurality of users who wear the display apparatus can sharethe visual fields and operation. Therefore, for example, a person whoperforms an action and a person who assists the action wear thehead-mounted-type display apparatus for conveyance of a large amount ofinformation on actions in a more understandable manner. Further,evaluation of the state of a user allows notification data to betransmitted at appropriate timing. A user can therefore assist anotheruser who performs work or any other type of act or readily andaccurately administer the state in which work or any other type of actis performed.

In the display system according to the aspect of the invention, theevaluation section may evaluate, on the basis of an action of the user,whether the state of the user corresponds to a state in whichnotification set in advance should be made, and when the evaluationsection determines that the state of the user corresponds to the statein which notification should be made, the control section may cause thecommunication section to transmit notification data to the anotherdisplay apparatus.

According to the aspect of the invention with this configuration,determination that the state of the user is the state in whichnotification should be made can be made, and notification data can betransmitted to another apparatus.

The display system according to the aspect of the invention may furtherinclude a recognition section that recognizes an input, and theevaluation section may evaluate the state of the user on the basis ofthe input recognized by the recognition section.

According to the aspect of the invention with this configuration, theeach of the display apparatus can recognize an input from the user oranother person to appropriately evaluate the state of the user. Further,the user or another person allows the display apparatus to recognize aninput to intentionally cause the display apparatus to transmitnotification data to another apparatus.

The display system according to the aspect of the invention may furtherinclude a voice processing section that detects voice, and therecognition section may recognize an input on the basis of the voicedetected by the voice processing section.

According to the aspect of the invention with this configuration, eachof the display apparatus can accept an input in the form of voice,whereby information can be more readily conveyed among the plurality ofdisplay apparatus.

In the display system according to the aspect of the invention, thecontrol section may transmit data representing the input recognized bythe recognition section on the basis of the voice detected by the voiceprocessing section to the another display apparatus.

According to the aspect of the invention with this configuration, acontent inputted in the form of voice to each of the display apparatuscan be notified to a user who wears another display apparatus.

The display system according to the aspect of the invention may furtherinclude a motion detection section that detects motion of the displaysection, and the recognition section may recognize an input on the basisof the motion detected by the motion detection section.

According to the aspect of the invention with this configuration, therecognition section can accept an input according to motion of thedisplay section or motion of the user who wears the display section.

In the display system according to the aspect of the invention, therecognition section may recognize an input on the basis of an imagecaptured by the imaging section.

According to the aspect of the invention with this configuration, therecognition section can recognize an input on the basis of a capturedimage and can therefore more readily accept an input.

In the display system according to the aspect of the invention, thecontrol section may perform a reproduction process of reproducing actionguidance data containing guidance information that gives guidance onactions, order information that specifies an order in accordance withwhich the actions are performed, and state information that shows astate of the performed actions, cause the display section to performdisplay in the reproduction process and updates, on the basis of aninput recognized by the recognition section, the state information inthe action guidance data being reproduced, and cause the communicationsection to transmit the action guidance data having undergone thereproduction process to the another display apparatus.

According to the aspect of the invention with this configuration, actionguidance can be given to a user who wears the display apparatus andperforms an action, and data containing records on the action performedin accordance with the guidance can be taken over and used by anotherdisplay apparatus.

In the display system according to the aspect of the invention, theimaging section may perform image capturing over a range that overlapswith a direction in which the user gazes.

According to the aspect of the invention with this configuration, animage in the direction in which the user gazes can be shown to anotheruser.

Another aspect of the invention relates to a head-mounted-type displayapparatus including a display section that allows a user to view animage and transmits an outside scene, a communication section thatcommunicates with another head-mounted-type display apparatus, animaging section that performs image capturing over a range that overlapswith a visual field of the user, an evaluation section that evaluates astate of the user, and a control section, and the control sectiontransmits visual field data on the basis of an image captured by theimaging section to the another display apparatus, causes thecommunication section to transmit notification data on the basis of aresult of the evaluation performed by the evaluation section to theanother display apparatus, and causes the display section to performdisplay on the basis of data transmitted from the another displayapparatus.

According to the aspect of the invention, data on the visual fields anddata on inputs can be shared with another display apparatus. Therefore,for example, in a configuration in which a plurality of users wear thedisplay apparatus, the visual fields and operation can be shared by theusers. In this case, a person who performs an action and a person whoassists the action wear the display apparatus for conveyance of a largeamount of information on actions in a more understandable manner.Further, evaluation of the state of a user allows notification data tobe transmitted at appropriate timing.

In the aspect of the invention, a head-mounted-type display apparatusincluding a display section that allows a user to view an image andtransmits an outside scene is caused to perform image capturing over arange that overlaps with a visual field of the user, evaluate a state ofthe user, transmit visual field data on the basis of captured image datato another head-mounted-type display apparatus, transmit notificationdata on the basis of a result of the evaluation of the state of the userto the another display apparatus, and cause the display section toperform display on the basis of data transmitted from the anotherdisplay apparatus.

According to the aspect of the invention with this configuration, dataon the visual fields and data on inputs can be shared with anotherdisplay apparatus. Therefore, for example, in a configuration in which aplurality of users wear the display apparatus, the visual fields andoperation can be shared by the users. In this case, a person whoperforms an action and a person who assists the action wear the displayapparatus for conveyance of a large amount of information on actions ina more understandable manner. Further, evaluation of the state of a userallows notification data to be transmitted at appropriate timing.

The invention can also be configured as a program executable by acomputer that controls a head-mounted-type display apparatus including adisplay section that allows a user to view an image and transmits anoutside scene. The program causes the computer to perform imagecapturing over a range that overlaps with a visual field of the user,evaluate a state of the user, transmit visual field data on the basis ofcaptured image data to another head-mounted-type display apparatus,transmit notification data on the basis of a result of the evaluation ofthe state of the user to the another display apparatus, and cause thedisplay section to perform display on the basis of data transmitted fromthe another display apparatus. The invention may also be configured as astorage medium that stores the program.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be described with reference to the accompanyingdrawings, wherein like numbers reference like elements.

FIG. 1 is a schematic configuration diagram of a display systemaccording to an embodiment.

FIG. 2 shows an example of the arrangement of the display system.

FIG. 3 is a descriptive diagram showing an exterior configuration of ahead mounted display.

FIG. 4 shows the configuration of an optical system of an image displaysection.

FIGS. 5A and 5B show the configurations of key parts of the imagedisplay section.

FIG. 6 is a functional block diagram of portions that form the HMD.

FIG. 7 is a diagrammatic view showing data stored in a storage section.

FIGS. 8A and 8B are diagrammatic views showing an example of theconfiguration of data stored in the storage section. FIG. 8A shows anexample of transmission destination setting data, and FIG. 8B shows anexample of the configuration of motion guidance data.

FIGS. 9A and 9B are flowcharts showing the action of the display system.

FIGS. 10A and 10B are flowcharts showing the action of the displaysystem.

FIGS. 11A to 11C show examples of display in the display system.

FIGS. 12A to 12D are descriptive diagrams showing a specific applicationexample of the invention.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

FIG. 1 is a schematic configuration diagram of a display system 1according to an embodiment to which the invention is applied.

The display system 1 includes a plurality of HMDs 100, and the HMDs 100are connected to one another in a communicatable manner via acommunication network 4. Each of the HMDs 100 is a display apparatusthat a user wears around the head and also called a head mounteddisplay. Each of the HMDs 100 is an optically transmissive HMD thatallows the user to not only view a virtual image but also directly viewan outside scene at the same time. In the following description, avirtual image that the HMD 100 allows the user to view is also called a“display image” for convenience. Further, outputting image lightgenerated on the basis of image data is also referred to as “displayingan image.”

Three HMDs 100 illustrated in FIG. 1 are worn by different users. Forexample, three users wear the HMDs 100 and are allowed to use them atthe same time. In the following description, the three HMDs 100 arecalled HMDs 100A, 100B, and 100C for convenience of the description. Thelocations where the HMDs 100A, 100B, and 100C are used are assumed to besites A, B, and C, respectively.

In the display system 1, users who perform actions wear the HMDs 100Band 100C, a user who instructs and administers actions wears the HMD100A, and transmission and reception of information for the instructionand administration are performed among the HMDs 100A, 100B, and 100C.The sites A, B, and C are free from geographical restrictions as long asthey are connectable to the communication network 4 and may be remotefrom each other or close to each other. In the following description, acase where the three HMDs 100A, 100B, and 100C are connected to thecommunication network 4 is shown, but no constraint is imposed on thenumber of HMDs 100 provided in the display system 1.

FIG. 2 shows an example of the arrangement of the HMDs 100A, 100B, and100C in the display system 1.

The HMDs 100B and 100C are arranged, for example, in the sites B and C,respectively, where workers perform work, in a work line FA in afactory. A worker UB who performs work in the site B wears the HMD 100B,and a worker UC who performs work in the site C wears the HMD 100C.

In the work line FA, targets OB, such as parts that are targets of work,are transported in the direction indicated by the arrows, and theworkers UB and UC perform instructed work on the targets OB.

An administrator UA, who administers the work line FA, wears and usesthe HMD 100A in the site A, which is a site for an administrator. Theadministrator UA uses the display system 1 to assist the work performedby the workers UB and UC, administer the progress of the work, andperform other types of action.

FIG. 2 shows an application example in which in a manufacturing factoryor any other place, an administrator wears an HMD 100, workers wear HMDs100, and the administrator assists work performed by workers,administers the progress of the work, administers the quality of aresult of the work, and performs other types of action, but a targetsituation in which the display system 1 can be used is not limited tothe manufacturing factory described above and the following applicationexamples 1 to 4 can be listed.

Example 1: In a work place accompanied by a warehouse, workers who movein the warehouse and pick up articles and an administrator wear HMDs100. The HMD 100 on each of the workers displays a guide path alongwhich the worker moves in the warehouse to guide the worker andindicates an article to be picked up. The work administrator wears anHMD 100 to administer the progress of the work and perform other typesof action.

Example 2: In a construction site, an HMD 100 on a worker displays theposition of a facility or any other object that the worker cannotdirectly view, such as an underground pipe, on the basis of designdrawings to guide the worker. A construction administrator wears an HMD100 to administer the progress of the construction.

Example 3: A user who wears an HMD 100 and moves receives movement pathguidance and route guidance. An administrator for the movement wears anHMD 100 and administers the situation of the movement of the user andthe position thereof.

Example 4: In a medical institute, a medical practitioner who wears anHMD 100 receives assistance in medical consultation, medicalexamination, or any other act. An assistant who wears an HMD 100 checksact performed by the medical practitioner, administers the progress ofthe act, and performs other types of action.

The communication network 4 is achieved by a variety of communicationlines, such as a public network, a dedicated line, a wirelesscommunication line including a mobile telephone line, a backbonecommunication line of any of the lines described above, or a combinationthereof, and the configuration of the communication network 4 is notlimited to a specific configuration. The communication network 4 may bea wide area communication network capable of connecting remote locationsto each other or a LAN (local area network) installed in a specificinstitution or building. The communication network 4 may include aserver device and a gateway device, a router device, or any othernetwork apparatus that connect the variety of communication linesdescribed above to each other. The communication network 4 may insteadbe formed of a plurality of communication lines. The HMDs 100A, 100B,and 100C wirelessly communicate with a wireless access point (not shown)or any other device that forms the communication network 4 to transmitand receive data to and from each other via the communication network 4.

The HMDs 100A, 100B, and 100C will be described on the assumption thatthey have the same configuration. When the HMDs 100A, 100B, and 100C arenot distinguished from each other, they are collectively called the HMD100.

The HMD 100 includes an image display section 20, which a commander whois a user wears around the head, and a control device 10, which controlsthe image display section 20. The image display section 20 is wornaround the user's head and allows the user to view a virtual image. Thecontrol device 10 also functions as a controller used by the user tooperate the HMD 100.

FIG. 3 is a descriptive diagram showing an exterior configuration of theHMD 100.

The image display section 20 is a wearable member worn around the user'shead and has a spectacle-like shape in the present embodiment. The imagedisplay section 20 includes a right holder 21, a right display driver22, a left holder 23, a left display driver 24, a right optical imagedisplay section 26, a left optical image display section 28, a rightcamera 61 (imaging section), a left camera 62 (imaging section), and amicrophone 63. The right optical image display section 26 and the leftoptical image display section 28 are so disposed as to be located infront of the right and left eyes of the user who wears the image displaysection 20. One end of the right optical image display section 26 andone end of the left optical image display section 28 are linked to eachother in a position corresponding to the portion between the eyes of theuser who wears the image display section 20.

The right holder 21 is a member extending from an end ER of the rightoptical image display section 26, which is the other end thereof, to aposition corresponding to a temporal region of the user who wears theimage display section 20. Similarly, the left holder 23 is a memberextending from an end EL of the left optical image display section 28,which is the other end thereof, to a position corresponding to anothertemporal region of the user who wears the image display section 20. Theright holder 21 and the left holder 23, which serve in the same manneras temples (bows) of spectacles do, hold the image display section 20around the user's head.

The right display driver 22 and the left display driver 24 are disposedon opposite sides of the head of the user who wears the image displaysection 20. The right display driver 22 and the left display driver 24are also simply called “display drivers” in a collective manner, and theright optical image display section 26 and the left optical imagedisplay section 28 are also simply called “optical image displaysections” in a collective manner.

The display drivers 22 and 24 include liquid crystal displays 241 and242 (hereinafter referred to as “LCDs 241 and 242”), projection systems251 and 252, which will be described later with reference to FIGS. 4 to6, and other components.

The right optical image display section 26 and the left optical imagedisplay section 28 include light guide plates 261 and 262 (FIG. 4) and alight control plate 20A. The light guide plates 261 and 262 are made,for example, of a light transmissive resin material and guide imagelight outputted from the display drivers 22 and 24 to the user's eyes.The light control plate 20A is a thin-plate-shaped optical element andso disposed as to cover the front side of the image display section 20,which is opposite the side where the user's eyes are present. The lightcontrol plate 20A can be a plate having light transmittance ofsubstantially zero, a nearly transparent plate, a plate that transmitslight but attenuates the amount of light, a plate that attenuates orreflects light of a specific wavelength, or any of other variety ofoptical components. Appropriate selection of optical characteristics(such as light transmittance) of the light control plate 20A allowsadjustment of the amount of external light externally incident on theright optical image display section 26 and the left optical imagedisplay section 28 and hence adjustment of visibility of a virtualimage. In the present embodiment, a description will be made of a casewhere the light control plate 20A is optically transmissive enough atleast to allow the user who wears the HMD 100 to view an outside scene.The light control plate 20A also protects the right light guide plate261 and the left light guide plate 262 and prevents the right lightguide plate 261 and the left light guide plate 262 from being damaged,dirt from adhering thereto, and other defects from occurring.

The light control plate 20A may be configured to be attachable to anddetachable from the right optical image display section 26 and the leftoptical image display section 28, or a plurality of types of lightcontrol plate 20A may be exchangeably attachable. The light controlplate 20A may even be omitted.

The right camera 61 is disposed on the front surface of the HMD 100 andin an end portion on the side facing the right holder 21. Similarly, theleft camera 62 is disposed on the front surface of the HMD 100 and in anend portion on the side facing the left holder 23. Each of the rightcamera 61 and the left camera 62 is a digital camera including a CCD, aCMOS device, or any other imaging device, an imaging lens, and othercomponents and may be a monocular camera or a stereoscopic camera.

Each of the right camera 61 and the left camera 62 captures an image ofat least part of an outside scene present on the front side of the HMD100, in other words, in the visual field direction of the user who wearsthe HMD 100. In another expression, at least one of the right camera 61and the left camera 62 performs image capturing over a range thatoverlaps with the user's visual field or in the direction thereof. Morespecifically, at least one of the right camera 61 and the left camera 62performs image capturing in the direction in which the user gazes. Theangle of view of each of the right camera 61 and the left camera 62 canbe set at an appropriate value and is, in the present embodiment, anangle of view that covers the outside viewed by the user through theright optical image display section 26 and the left optical imagedisplay section 28, as will be described later. Further, it is morepreferable that the image capturing range of each of the right camera 61and the left camera 62 is so set that the camera can perform imagecapturing over the entire visual field of the user through the lightcontrol plate 20A.

Each of the right camera 61 and the left camera 62 performs imagecapturing under the control of an imaging control section 161 (FIG. 6),which is provided in a control section 140, and outputs captured imagedata to the imaging control section 161.

Distance sensors 64 are disposed in a portion at the boundary betweenthe right optical image display section 26 and the left optical imagedisplay section 28. In a state in which the user wears the image displaysection 20, the positions of the distance sensors 64 are roughly at themiddle between the user's eyes in the horizontal direction but above theuser's eyes in the vertical direction.

The distance sensors 64 detect the distance to a target object undermeasurement located in a preset measurement direction. Each of thedistance sensors 64 includes, for example, an LED, a laser diode, or anyother light source and a light receiver that receives light emitted fromthe light source and reflected off the target object under measurement.In this case, the distance sensors 64 perform the distance measurementbased on triangulation or time difference under the control of thecontrol section 140. Each of the distance sensors 64 may instead includea sound source that emits an ultrasonic wave and a detector thatreceives the ultrasonic wave reflected off a target object undermeasurement. In this case, the distance sensors 64 perform the distancemeasurement based on the difference in time spent until the ultrasonicwave is reflected under the control of the control section 140. Each ofthe distance sensors 64 may instead include a light source and a lightreceiver or a sound source and a detector, and the control section 140may perform the distance measurement.

The measurement direction of the distance sensors 64 in the presentembodiment is the frontward direction from the END 100 and coincideswith the image capturing direction of the right camera 61 and the leftcamera 62.

FIG. 4 is a key portion plan view showing the configuration of anoptical system provided in the image display section 20. FIG. 4 shows auser's left eye LE and right eye RE for the description.

The left display driver 24 includes a left backlight 222, which has anLED or any other light source and a diffuser plate, and the left LCD242, which is a transmissive LCD and disposed in the light path of lighthaving exited out of the diffuser plate of the left backlight 222. Theleft display driver 24 further includes the left projection system 252,which includes a lens group and other components that guide image lightL having passed through the left LCD 242. The left LCD 242 is atransmissive liquid crystal panel having a plurality of pixels arrangedin a matrix.

The left projection system 252 has a collimation lens that converts theimage light L having exited out of the left LCD 242 into a parallelizedlight flux. The image light L, which is the parallelized light fluxhaving been converted by the collimation lens, enters the left lightguide plate 262. The left light guide plate 262 is a prism having aplurality of reflection surfaces that reflect the image light L, and theimage light L is reflected multiple times in the left light guide plate262 and then guided to the side where the left eye LE is present. Theleft light guide plate 262 has a half-silvered mirror 262A (reflectionsurface) formed thereon and disposed in front of the left eye LE.

The image light L reflected off the half-silvered mirror 262A exits outof the left optical image display section 28 toward the left eye LE, andthe image light L forms an image on the retina of the left eye LE andallows the user to view the image.

The right display driver 22 is a bilaterally symmetric display driver ofthe left display driver 24. The right display driver 22 includes a rightbacklight 221, which has an LED or any other light source and a diffuserplate, and the right LCD 241, which is a transmissive LCD and disposedin the light path of light having exited out of the diffuser plate ofthe right backlight 221. The right display driver 22 further includesthe right projection system 251, which includes a lens group and othercomponents that guide image light L having passed through the right LCD241. The right LCD 241 is a transmissive liquid crystal panel having aplurality of pixels arranged in a matrix.

The right projection system 251 has a collimation lens that converts theimage light L having exited out of the right LCD 241 into a parallelizedlight flux. The image light L, which is the parallelized light fluxhaving been converted by the collimation lens, enters the right lightguide plate 261. The right light guide plate 261 is a prism having aplurality of reflection surfaces that reflect the image light L, and theimage light L is reflected multiple times in the right light guide plate261 and then guided to the side where the right eye RE is present. Theright light guide plate 261 has a half-silvered mirror 261A (reflectionsurface) formed thereon and disposed in front of the right eye RE.

The image light L reflected off the half-silvered mirror 261A exits outof the right optical image display section 26 toward the right eye RE,and the image light L forms an image on the retina of the right eye REand allows the user to view the image.

On the user's right eye RE are incident the image light L having beenreflected off the half-silvered mirror 261A and outside light OL havingpassed through the light control plate 20A. On the user's left eye LEare incident the image light L having been reflected off thehalf-silvered mirror 262A and outside light OL having passed through thelight control plate 20A. The HMD 100 thus causes the image light L,which carries an internally processed image, and the outside light OL tobe superimposed on each other and incident on the user's eyes, wherebythe user sees an outside scene through the light control plate 20A andviews the image carried by the image light L with the image superimposedon the outside scene. The HMD 100 thus functions as a see-through-typedisplay apparatus.

The left projection system 252 and the left light guide plate 262 arealso collectively called a “left light guide unit,” and the rightprojection system 251 and the right light guide plate 261 are alsocollectively called a “right light guide unit.” The configuration ofeach of the right light guide unit and the left light guide unit is notlimited to the example described above and can be arbitrarily configuredas long as the image light is used to form a virtual image in front ofthe user's eyes. For example, a diffraction grating may be used, or asemitransparent reflection film may be used.

The image display section 20 is connected to the control device 10 via aconnection section 40. The connection section 40 includes a body cord48, which is connected to the control device 10, a right cord 42, a leftcord 44, and a linkage member 46. The right cord 42 and the left cord 44are two cords into which the body cord 48 bifurcates. The right cord 42is inserted into an enclosure of the right holder 21 through alengthwise end portion AP of the right holder 21 and connected to theright display driver 22. Similarly, the left cord 44 is inserted into anenclosure of the left holder 23 through a lengthwise end portion AP ofthe left holder 23 and connected to the left display driver 24.

The linkage member 46 is disposed at the point where the body cord 48bifurcates into the right cord 42 and the left cord 44 and has a jackfor connecting an earphone plug 30. A right earphone 32 and a leftearphone 34 extend from the earphone plug 30. The microphone 63 isprovided in a position in the vicinity of the earphone plug 30. Anintegrated single cord form the portion from the earphone plug 30 to themicrophone 63 and bifurcates at the microphone 63 into two cordsconnected to the right earphone 32 and the left earphone 34,respectively.

The microphone 63 is so disposed that a voice collection portion of themicrophone 63 faces the directions of the user's sight lines, as shown,for example, in FIG. 3, and the microphone 63 collects voice and outputsa voice signal to a voice processing section 187 (FIG. 6). Themicrophone 63 may, for example, a monophonic microphone or astereophonic microphone. The microphone 63 may be a directionalmicrophone or an omni-directional microphone.

Each of the right cord 42, the left cord 44, and the body cord 48 may beany cord capable of transporting digital data and can be formed, forexample, of a metal cable or an optical fiber. The right cord 42 and theleft cord 44 may be integrated into a single cord.

The image display section 20 and the control device 10 transport avariety of signals to each other via the connection section 40.Connectors (not shown) that fit into each other are provided at the endof the body cord 48 on the side opposite the linkage member 46 and at anend of the control device 10. Causing the connector at the body cord 48and the connector at the control device 10 to fit into each other anddetach from each other allows the control device 10 and the imagedisplay section 20 to be connected to each other and disconnected fromeach other.

The control device 10 controls the HMD 100. The control device 10 has agroup of switches including a finalizing key 11, a lighting section 12,a display switch key 13, a luminance switch key 15, a direction key 16,a menu key 17, and a power switch 18. The control device 10 furtherincludes a trackpad 14, operated by the user with a finger.

The finalizing key 11 detects pressing operation and outputs a signalthat finalizes the content of the operation performed on the controldevice 10. The lighting section 12 includes a light source, such as anLED (light emitting diode), and notifies the user of the state of actionof the HMD 100 (whether the HMD 100 is powered on or off, for example)in the form of the lighting state of the light source. The displayswitch key 13 outputs a signal that instructs, for example, switching ofan image display mode from one to another in accordance with pressingoperation performed on the display switch key 13.

The trackpad 14 has an operation surface that detects contact operationand outputs an operation signal according to the operation performed onthe operation surface. A method for detecting operation performed on theoperation surface is not limited to a specific method and can, forexample, be an electrostatic method, a pressure detection method, or anoptical method. The luminance switch key 15 outputs a signal thatinstructs an increase or a decrease in the luminance of an imagedisplayed by the image display section 20 in accordance with pressingoperation performed on the luminance switch key 15. The direction key 16outputs an operation signal in accordance with pressing operationperformed on a key corresponding to any of the upward, downward,rightward, and leftward directions. The power switch 18 switches thepower on/off state of the HMD 100 from one to the other.

FIGS. 5A and 5B show the configurations of key parts of the imagedisplay section 20. FIG. 5A is a perspective view of the key parts in acase where the image display section 20 is viewed from the user's headside, and FIG. 5B is a descriptive diagram of the angles of view of theright camera 61 and the left camera 62. In FIG. 5A, the right cord 42and the left cord 44, which are tied to the image display section 20,and other components are omitted.

FIG. 5A shows one side of the image display section 20, that is, theside thereof facing the user's head, in other words, the side visible tothe user's right eye RE and left eye LE. In another expression, FIG. 5Ashows the rear side of the right optical image display section 26 andthe left optical image display section 28.

In the HMD 100, the half-silvered mirrors 261A and 262A reflect theimage light to the user's eyes to form a display region, as shown inFIG. 5A. In the HMD 100, the half-silvered mirrors 261A and 262A reflectthe image light to allow the user's eyes to recognize a virtual image.Therefore, in the present embodiment, the display region is not thehalf-silvered mirrors 261A and 262A themselves but is a region where theuser perceives the image light reflected off the half-silvered mirrors261A and 262A. In a case where the image display section 20 has aconfiguration in which images are actually formed on the half-silveredmirrors 261A and 262A, the half-silvered mirrors 261A and 262A serve asthe display region. The display region is a region which corresponds tothe right LCD 241 and the left LCD 242 and where the user views imagesdisplayed on the LCDs 241 and 242. For example, when the right LCD 241and the left LCD 242 display images over entire regions thereof whereimages can be displayed (image displayable regions), the user is allowedto view an image of the size of the entire display region.

As described above, the half-silvered mirror 261A, which irradiates theuser's right eye RE with the image light, and the half-silvered mirror262A, which irradiates the user's left eye LE with the image light,allow the user to view a roughly rectangular display image. Further, theentire right optical image display section 26 and left optical imagedisplay section 28 including the half-silvered mirrors 261A and 262Atransmits outside light, as described above. The user therefore views anoutside scene that passes through the entire right optical image displaysection 26 and left optical image display section 28 and a display imageformed by the image light reflected off the half-silvered mirrors 261Aand 262A with the outside scene and the display image superimposed oneach other.

The right camera 61 is disposed at an end portion facing the rightholder 21 and oriented forward with respect to the image display section20, and the left camera 62 is disposed at an end portion facing the leftholder 23, as described above. The distance sensors 64 are disposed atthe middle between the right optical image display section 26 and theleft optical image display section 28 and oriented forward.

FIG. 5B diagrammatically shows the positions of the right camera 61, theleft camera 62, and the distance sensors 64 in a plan view along withthe user's right eye RE and left eye LE. In FIG. 5B, CR represents theangle of view (imaging range) of the right camera 61, and CL representsthe angle of view (imaging range) of the left camera 62. The angles ofview CR and CL shown in FIG. 5B are angles of view in the horizontaldirection, and it is noted that actual angles of view of the rightcamera 61 and the left camera 62 extend also in the upward/downwarddirection, as in the case of a typical digital camera.

The angles of view CR and CL are bilaterally symmetric with respect tothe center position of the image display section 20, and each of theangles of view CR and CL includes a direction passing through a pointright in front of the center position of the image display section 20.The angles of view CR and CL therefore overlap with each other in frontof the center position of the image display section 20.

For example, when a target OB is located in front of the image displaysection 20, the target OB falls within both the angles of view CR andCL, as shown in FIG. 5B. The target OB is therefore displayed in both animage captured with the right camera 61 and an image captured with theleft camera 62. In this case, when the user gazes the target OB, theuser's sight lines are directed to the target OB, as indicated by thereference characters RD and LD in FIG. 5B. In general, it is believedthat the angle of field of a person is about 200 degrees in thehorizontal direction and about 125 degrees in the vertical direction.Within these ranges, an effective field of view where a person hasexcellent information reception capability extends over a horizontalrange of about 30 degrees and a vertical range of about 20 degrees.Further, it is believed that a stable field of fixation where a point offixation at which a person gazes in a quick, stable manner extends overa horizontal range from about 60 to 90 degrees and a vertical range fromabout 45 to 70 degrees.

Therefore, when a point of fixation coincides with the target OB, theeffective field of view extends over about 30 degrees in the horizontaldirection and about 20 degrees in the vertical direction, the stablefield of fixation extends over about 60-90 degrees in the horizontaldirection and about 45-70 degrees in the vertical direction, and theangle of field is about 200 degrees in the horizontal direction andabout 125 degrees in the vertical direction, over portions on both sidesof the sight lines RD and LD.

Further, an actual field of view viewed by the user who wears the HMD100 through the image display section 20 and further through the rightoptical image display section 26 and the left optical image displaysection 28 is called an actual field of view (FOV). In the configurationof the present embodiment, the actual field of view corresponds to afield of view actually viewed by the user through the right opticalimage display section 26 and the left optical image display section 28(field of view VR, which will be described later). The actual field ofview is narrower than the angle of field and the stable field offixation, which have been described with reference to FIG. 5B, but widerthan the effective field of view.

Each of the right camera 61 and the left camera 62 is preferably capableof image capturing over a range wider than the user's field of view.Specifically, the sum of the angles of view CR and CL is preferablywider than at least the user's effective field of view. More preferably,the sum of the angles of view CR and CL is wider than the user's actualfield of view. Still more preferably, the sum of the angles of view CRand CL is wider than the user's stable field of fixation. Mostpreferably, the sum of the angles of view CR and CL is wider than theuser's angle of the field.

To this end, the right camera 61 and the left camera 62 are so disposedthat the angles of view CR and CL overlap with each other in front ofthe image display section 20, as shown in FIG. 5B. Each of the rightcamera 61 and the left camera 62 may be formed of a wide-angle camera.That is, each of the right camera 61 and the left camera 62 may includewhat is called a wide-angle lens as the imaging lens for image capturingover a wide angle of view. The wide-angle lens may include a lens calleda super-wide-angle lens or a semi-wide-angle lens or may be afixed-focal-length lens or a zoom lens, or each of the right camera 61and the left camera 62 may include a lens group formed of a plurality oflenses. Further, the angle of view CR of the right camera 61 may not beequal to the angle of view CL of the left camera 62. Moreover, theimaging direction of the right camera 61 is not necessarily completelyparallel to the imaging direction of the left camera 62. The rightcamera 61 and the left camera 62 only need to perform image capturing insuch a way that the combination of an image captured with the rightcamera 61 and an image captured with the left camera 62 is wider thanthe user's field of view.

In FIG. 53, the reference character 64A denotes the detection directionof the distance sensors 64. In the present embodiment, the distancesensors 64 are configured to be capable of detecting the distance fromthe center position of the image display section 20 to an object locatedin front thereof and, for example, detects the distance to the targetOB. Since the user who wears the HMD 100 orients the head in a gazingdirection, a target to be gazed is considered to be located in front ofthe image display section 20. The distance sensors 64, which aredisposed at the center of the image display section 20, can thereforedetect the distance to a target gazed by the user, provided that thedetection direction 64A coincides with the forward direction withrespect to the image display section 20.

Further, sight line sensors 68 are disposed on the user's side of theimage display section 20, as shown in FIG. 5A. The sight line sensors 68are provided as a pair in correspondence with the user's right eye REand left eye LE in positions at the middle between the right opticalimage display section 26 and the left optical image display section 28.The sight line sensors 68 are formed, for example, of a pair of camerasthat captures images of the user's right eye RE and left eye LE. Thesight line sensors 68 perform image capturing under the control of thecontrol section 140 (FIG. 6), and the control section 140 detects, fromcaptured image data, light reflected off the eyeball surfaces of theright eye RE and the left eye LE and images of the pupils to identifythe directions of the sight lines.

FIG. 6 is a functional block diagram of portions that form the HMD 100.

The HMD 100 includes an interface 125, which connects a variety ofexternal apparatus OA, which serve as content supply sources, to the HMD100. The interface 125 can be an interface that supports wiredconnection, such as a USB interface, a micro-USB interface, and a memorycard interface, and may instead be formed of a wireless communicationinterface. The external apparatus OA are each an image supply apparatusthat supplies the HMD 100 with images and are, for example, a personalcomputer (PC), a mobile phone terminal, and a mobile game console.

The control device 10 includes the control section 140, an inputinformation acquisition section 110, a storage section 120, atransmitter (Tx) 51, and a transmitter (Tx) 52.

The input information acquisition section 110 is connected to anoperation section 135. The operation section 135 includes the trackpad14, the direction key 16, the power switch 18, and other componentsdescribed above, and the input information acquisition section 110acquires an input content on the basis of a signal inputted through theoperation section 135. The control device 10 further includes a powersupply section (not shown) that supplies the portions in the controldevice 10 and the image display section 20 with electric power.

The storage section 120 is a nonvolatile storage device and stores avariety of computer programs and data associated with the programs. Thestorage section 120 may further store data on still images and motionimages to be displayed in the image display section 20.

The storage section 120 further stores setting data 121. The settingdata 121 contains a variety of setting values used by the controlsection 140. The setting values contained in the setting data 121 may bevalues having been inputted in advance through operation of theoperation section 135 or may be values received from any of the externalapparatus OA or any other device (not shown) via a communication section117 or the interface 125 and then stored.

The storage section 120 further stores a variety of data transmitted andreceived by the HMDs 100A, 100B, and 100C.

FIG. 7 diagrammatically shows data stored in the storage section 120.

The storage section 120 stores the setting data 121 and transmissiondestination setting data 122. The storage section 120 has a transmissiondata storage section 126 and a reception data storage section 127 as astorage region where data is stored. The transmission data storagesection 126 and the reception data storage section 127 can be providedby virtual or logical partition of the storage region and therefore donot need to be classified, for example, as hardware.

The transmission data storage section 126 stores data transmitted by theHMD 100 to another HMD 100. That is, data stored in the transmissiondata storage section 126 is transmitted via the communication section117 to another HMD 100. On the other hand, the reception data storagesection 127 stores data received through the communication section 117.The data received through the communication section 117 is datatransmitted from another HMD 100 over the communication network 4.

The transmission data storage section 126 stores visual field data 131,notification data 132, and action guidance data 133 as data to betransmitted. The reception data storage section 127 stores visual fielddata 131, notification data 132, and action guidance data 133 receivedthrough the communication section 117.

The visual field data 131, the notification data 132, and the actionguidance data 133 stored in the transmission data storage section 126may be identical to the visual field data 131, the notification data132, and the action guidance data 133 stored in the reception datastorage section 127. The data described above stored in the transmissiondata storage section 126 may differ from the data stored in thereception data storage section 127 in terms of header, attribute, dataformat, and other factors.

The visual field data 131 is part of image data captured with the rightcamera 61 and the left camera 62, that is, data cut therefrom or imagedata containing the entire captured image data. That is, the visualfield data 131 corresponds to an image of the outside that the userlooks at, that is, an image of the visual field. The visual field data131 may contain still image data, motion image (video image) data, andvoice data. The visual field data 131 may instead contain voice data butno image data. The HMD 100 can transmit an image of the outside (visualfield) that the user looks at by transmitting the visual field data 131.Further, the HMD 100 can recreate the visual field that the user ofanother HMD 100 looks at by reproducing and displaying the visual fielddata 131 transmitted from the other HMD 100.

The notification data 132 contains data representing a content inputtedin the HMD 100. The input to the HMD 100 is, for example, an inputthrough operation of the control device 10, a voice input using voicerecognition, an input using knock operation on the image display section20, a gesture input achieved by placing a hand or any other indicatingtool in the imaging ranges of the right camera 61 and the left camera62, as will be described later. The HMD 100 can transmit the inputcontent to another HMD 100 by transmitting the notification data 132.Further, the HMD 100 allows the user to grasp the content of an input inanother HMD 100 by performing display based on the notification data 132transmitted from another HMD 100.

FIGS. 8A and 8B are diagrammatic views showing an example of theconfiguration of the data stored in the storage section 120. FIG. 8Ashows an example of the transmission destination setting data 122, andFIG. 8B shows an example of the configuration of the motion guidancedata 133.

The transmission destination setting data 122 is data that specifies thedestination to which the HMD 100 transmits data. In the display system 1shown in FIG. 1, the HMDs 100A, 100B, and 100C can set differentconstraints on the destination to which data is transmitted. Forexample, the HMD 100A is worn and used by the administrator UA, and theHMDs 100E and 100C are worn by the workers UB and UC. The destination towhich data is transmitted can be set in accordance with whether the userof the HMD is the administrator or a worker. In the transmissiondestination setting data 122, an ID that identifies an HMD 100 that isthe transmission destination and the type of data to be transmitted areset in relation to each other.

FIG. 8A shows an example of the transmission destination setting data122 in the HMD 100B used by the worker UB. The HMD 100B can transmit thevisual field data 131 and the notification data 132 to the HMD 100A (ID:0001 in FIG. 8A). The HMD 100B can transmit only the notification data132 to the HMD 100C (ID: 0003 in FIG. 8A).

As a similar setting, for example, the transmission destination settingdata 122 stored in the HMD 100A is so set that notification data can betransmitted to the HMDs 100B and 100C. Further, for example, in thetransmission destination setting data 122 stored in the HMD 100C, thevisual field data 131 and the notification data 132 can be transmittedto the HMD 100A and the notification data 132 can be transmitted to theHMD 100B.

The transmission destination setting data 122 may instead be data thatmakes setting of part of data stored and transmitted by the HMD 100. Forexample, the motion guidance data 133, which will be described later,can be configured not to be under constraint set by the transmissiondestination setting data 122 but to be transmittable to another HMD 100in the display system 1.

The motion guidance data 133 is data for displaying motion guidance oroutputting voice to the user who wears an HMD 100. The motion guidancedata 133 contains guidance information 133 a, which gives guidance onthe contents of actions, order information 133 b, which specifies theorder in accordance with which the actions are performed, and stateinformation 133 c, which shows the state of the performed actions, asshown in FIG. 8B.

The action performing order shown by the order information 133 b is arelative action order, and when the number of actions contained in theaction guidance data 133 is one, no order needs to be set in relationthereto.

Each action contained in the action guidance data 133 is given an item(title). In the example shown in FIG. 8B, an item that also serves asthe order information 133 b is given as follows: work 1, work 2, . . . .An action refers to an act or conduct performed by a user and may be abody motion of the user or may include thought and judgment. One actionin the action guidance data 133 may in practice correspond to a sequenceincluding a plurality of actions.

The guidance information 133 a contains a text, a still image, videoimages, voice, and other types of information outputted to the user whoperforms an action to describe or otherwise convey the content of theaction.

The state information 133 c is data on the history of each performedaction contained in the action guidance data 133. For example, the stateinformation 133 c is data on a progress history and a performing historyof the guidance information 133 a. In the example shown in FIG. 83, thestate information 133 c is set to show “complete” representing that anaction is complete, “under execution” representing a state in which anaction has been initiated but has not been completed, and “not executed”representing that an action has not been initiated.

The state information 133 c changes with the progress of an actionperformed by the user. The state information 133 c is updated inaccordance with the progress of an action under the control of thecontrol section 140, as will be described later.

To the control section 140 are connected a three-axis sensor 113, a GPS115, the communication section 117, and a voice recognition section 114.The three-axis sensor 113 is a three-axis acceleration sensor, and thecontrol section 140 acquires a detection value from the three-axissensor 113. The three-axis sensor 113 allows the control section 140 todetect the motion of the control device 10, for example, detectoperation of swinging or otherwise moving the control device 10. Thethree-axis sensor 113 may be replaced with a nine-axis sensor. In thiscase, the control section 140 can acquire detection values from athree-axis acceleration sensor, a three-axis angular velocity sensor,and a three-axis terrestrial magnetism sensor to detect, for example,the attitude, orientation, and motion, of the control device 10.

The GPS 115 includes an antenna (not shown) and receives a GPS (globalpositioning system) signal to calculate the current position of thecontrol device 10. The GPS 115 outputs the current position and thecurrent time determined on the basis of the GPS signal to the controlsection 140. The GPS 115 may further have a function of acquiring thecurrent time on the basis of information contained in the GPS signal tocorrect the time measured by the control section 140.

The communication section 117 performs wireless data communication thatcomplies with wireless THAN (WiFi (registered trademark)), Miracast(registered trademark), Bluetooth (registered trademark), or any otherstandard.

When the external apparatus OA is wirelessly connected to thecommunication section 117, the control section 140 acquires content datavia the communication section 117 and causes the image display section20 to display an image. On the other hand, when the external apparatusOA is wired to the interface 125, the control section 140 acquirescontent data via the interface 125 and causes the image display section20 to display an image. The communication section 117 and the interface125 function as a data acquisition section DA, which acquires contentdata from the external apparatus OA.

The control section 140 includes a CPU (not shown) that executes aprogram, a RAM (not shown) that temporarily stores the program executedby the CPU and data, and a ROM (not shown) that stores a basic controlprogram executed by the CPU and data in a nonvolatile manner. Thecontrol section 140 reads and executes the computer programs stored inthe storage section 120 to function as an operating system (OS) 150 andan image processing section 160. The control section 140 furtherfunctions as the imaging control section 161, an input recognitionsection 162 (recognition section), an AR display control section 163, astate detection section 164, a communication control section 165, anevaluation section 166, the voice processing section 187, and a displaycontrol section 190.

The image processing section 160 acquires an image signal contained in acontent. The image processing section 160 separates a vertical syncsignal. VSync, a horizontal sync signal HSync, and other sync signalsfrom the acquired image signal. Further, the image processing section160 produces a clock signal PCLK, for example, by using a PLL (phaselocked loop) circuit (not shown) in accordance with the cycles of theseparated vertical sync signal VSync and horizontal sync signal HSync.The image processing section 160 converts the analog image signal fromwhich the sync signals are separated into a digital image signal, forexample, by using an A/D conversion circuit (not shown). The imageprocessing section 160 then stores the converted digital image signal asimage data on a target image (Data in FIG. 6) in the RAM in the controlsection 140 on a frame basis. The image data is, for example, RGB data.

The image processing section 160 may perform, as required, resolutionconversion in which the resolution of the image data is converted intoresolution suitable for the right display driver 22 and the left displaydriver 24. The image processing section 160 may further perform imageadjustment in which the luminance and chroma of the image data areadjusted, 2D/3D conversion in which 2D image data is created from 3Dimage data or 3D image data is created from 2D image data.

The image processing section 160 transmits the clock signal PCLK, thevertical sync signal VSync, the horizontal sync signal HSync, and theimage data Data stored in the RAM via the transmitters 51 and 52. Eachof the transmitters 51 and 52 functions as a transceiver for serialtransportation between the control device 10 and the image displaysection 20. The image data Data transmitted via the transmitter 51 isalso called “image data for the right eye,” and the image data Datatransmitted via the transmitter 52 is also called “image data for theleft eye.”

The display control section 190 produces control signals that controlthe right display driver 22 and the left display driver 24, and thecontrol signals control the right display driver 22 and the left displaydriver 24 to cause them to produce and output image light. Specifically,the display control section 190 controls a right LCD control section 211to cause it to drive the right LCD 241 or not and controls a rightbacklight control section 201 to cause it to drive the right backlight221 or not. The display control section 190 further controls a left LCDcontrol section 212 to cause it to drive the left LCD 242 or not andcontrols a left backlight control section 202 to cause it to drive theleft backlight 222 or not.

The voice processing section 187 acquires a voice signal contained inthe content, amplifies the acquired voice signal, and outputs theamplified voice signal to the right earphone 32 and the left earphone34. The voice processing section 187 further acquires voice collectedthrough the microphone 63 and converts the collected voice into digitalvoice data. The voice processing section 187 may perform presetprocessing on the digital voice data.

The image display section 20 includes the right camera 61, the leftcamera 62, and the distance sensors 64 described above. The imagedisplay section 20 further includes an interface 25, the right displaydriver 22, the left display driver 24, the right light guide plate 261as the right optical image display section 26, the left light guideplate 262 as the left optical image display section 28, a nine-axissensor 66, and the sight line sensors 68.

The nine-axis sensor 66 (motion detection section) is a motion sensorthat detects acceleration (three axes), angular velocity (three axes),and terrestrial magnetism (three axes). The control section 140 candetect motion of the head of the user who wears the image displaysection 20 around the head on the basis of a detection value from thenine-axis sensor 66. For example, the control section 140 can estimatethe magnitude of inclination of the image display section 20 and theorientation of the inclination on the basis of a detection value fromthe nine-axis sensor 66.

The interface 25 includes a connector to which the right cord 42 and theleft cord 44 are connected. The interface 25 outputs the clock signalPCLK, the vertical sync signal VSync, the horizontal sync signal HSync,and the image data Data transmitted from the transmitters 51 and 52 tocorresponding receivers (Rx) 53 and 54. The interface 25 further outputsthe control signals transmitted from the display control section 190 tothe corresponding receivers 53, 54, right backlight control section 201,or left backlight control section 202.

The interface 25 is an interface that connects the right camera 61, theleft camera 62, the distance sensors 64, the nine-axis sensor 66, andthe sight line sensors 68 to the control section 140. Image datacaptured with the right camera 61 and the left camera 62, a result ofdetection performed by the distance sensors 64, a result of detection ofacceleration (three axes), angular velocity (three axes), andterrestrial magnetism (three axes) from the nine-axis sensor 66, and aresult of detection performed by the sight line sensors 68 are sent tothe control section 140 via the interface 25.

The right display driver 22 includes the right backlight 221, the rightLCD 241, and the right projection system 251 described above. The rightdisplay driver 22 further includes the receiver 53, the right backlight(BL) control section 201, which controls the right backlight (BL) 221,and the right LCD control section 211, which drives the right LCD 241.

The receiver 53 operates as a receiver corresponding to the transmitter51 and performs serial transportation between the control device 10 andthe image display section 20. The right backlight control section 201drives the right backlight 221 on the basis of the inputted controlsignal. The right LCD control section 211 drives the right LCD 241 onthe basis of the clock signal PCLK, the vertical sync signal VSync, thehorizontal sync signal HSync, and the image data Data for the right eye,which are inputted via the receiver 53.

The left display driver 24 has the same configuration as that of theright display driver 22. The left display driver 24 includes the leftbacklight 222, the left LCD 242, and the left projection system 252described above. The left display driver 24 further includes thereceiver 54, the left backlight control section 202, which drives theleft backlight 222, and the left LCD control section 212, which drivesthe left LCD 242.

The receiver 54 operates as a receiver corresponding to the transmitter52 and performs serial transportation between the control device 10 andthe image display section 20. The left backlight control section 202drives the left backlight 222 on the basis of the inputted controlsignal. The left LCD control section 212 drives the left LCD 242 on thebasis of the clock signal PCLK, the vertical sync signal VSync, thehorizontal sync signal HSync, and the image data Data for the left eye,which are inputted via the receiver 54.

The right backlight control section 201, the right LCD control section211, the right backlight 221, and the right LCD 241 are alsocollectively referred to as a right “image light generation unit.”Similarly, the left backlight control section 202, the left LCD controlsection 212, the left backlight 222, and the left LCD 242 are alsocollectively referred to as a left “image light generation unit.”

The imaging control section 161 controls the right camera 61 and theleft camera 62 to cause them to perform image capturing for capturedimage data acquisition. The imaging control section 161 may cause onlyone of the right camera 61 and the left camera 62 to perform imagecapturing or may cause both the right camera 61 and the left camera 62to perform image capturing.

The input recognition section 162 detects and recognizes an input to theHMD 100. The input recognition section 162 detects operation performedon the operation section 135 on the basis of a signal inputted from theinput information acquisition section 110. Further, when the voiceprocessing section 187 analyzes voice collected through the microphone63, and a result of the analysis shows that the voice is a voice commandset in advance, the input recognition section 162 detects the input ofthe voice command. Further, when the input recognition section 162detects an image of an indicating tool, such as the user's hand, fromimages captured with the right camera 61 and/or the left camera 62 anddetermines that the position, orientation, or motion of the indicatingtool corresponds to a gesture set in advance, the input recognitionsection 162 detects the gesture input. Further, when the inputrecognition section 162 determines that a pattern of detection signalsfrom the nine-axis sensor 66 provided in the image display section 20corresponds to operation of knocking the image display section 20, theinput recognition section 162 detects that the knock operation on theimage display section 20 as an input. Further, when the inputrecognition section 162 detects an image of an input marker, such as atwo-dimensional code or a barcode, from images captured with the rightcamera 61 and/or the left camera 62, the input recognition section 162detects the marker reading input. When the input recognition section 162detects an indicating tool or a marker from image data captured with theright camera 61 and/or the left camera 62, the input recognition section162 may detect that the indicating tool or marker corresponding to thedirection of the user's sight lines detected with the sight line sensors68.

The HMD 100 may include a foot switch (not shown) operated by the userwith a foot. The foot switch may be wired to the control section 140 orthe operation section 135 or may be connected to the communicationsection 117 over wireless communication, such as Bluetooth (registeredtrademark). In this case, the input recognition section 162 detectsoperation of the foot switch and recognizes the operation as an input.

The AR display control section 163 reads data stored in the storagesection 120 and controls the image processing section 160 and thedisplay control section 170 to cause the image display section 20 todisplay an image or a text. The AR display control section 163 mayAR-display, 3D-display (stereoscopically display), or 2D-display(planarly display) the image or the text in a position that coincideswith the position of a target in an actual space present in the user'sfield of view. Further, the AR display control section 163 may controlthe voice processing section 187 on the basis of the data read from thestorage section 120 to cause the right earphone 32 and the left earphone34 to output voice.

Specifically, the AR display control section 163 displays a still imageor video images in the visual field of a user of another HMD 100 on thebasis of the visual field data 131 stored in the reception data storagesection 127. Voice may be further outputted in this process.

The AR display control section 163 further reads the action guidancedata 133 in the reception data storage section 127 and displays a stillimage, motion images, a text, or any other type of information thatgives guidance on an action on the basis of the action guidance data133. In this process, the AR display control section 163 may display thecontents of guidance information 133 a. corresponding to a plurality ofactions in the form of a list in which the contents of information arearranged in accordance with the order specified by the order information133 b. The AR display control section 163 may instead display thecontents of guidance information 133 a by switching one content toanother or one set of a preset number of contents to another set inaccordance with the order specified by the order information 133 b.

The state detection section 164 detects the state of a performed action(work) corresponding to the guidance information 133 a in the actionguidance data 133 during a period for which the AR display controlsection 163 performs display based on the action guidance data 133.

The performed action state detection performed by the state detectionsection 164 is performed, for example, by analysis of image datacaptured with the right camera 61 and/or the left camera 62. In thiscase, the state detection section 164 detects at least one of a targeton which an action is performed, a gadget or a tool used to perform theaction, the user's body part, or any other object from the capturedimage data. The state detection section 164 generates data on the stateof the performed action on the basis of the position, size, color,shape, and other factors of the target on which the action is performed,the gadget or the tool used to perform the action, the user's body part,or any other object.

Instead, in the state in which the AR display control section 163displays an item in the action guidance data 133, the state detectionsection 164 may detect the state of a performed action corresponding tothe item on the basis of an input detected by the input recognitionsection 162.

When the input recognition section 162 detects an indicating tool or amarker from image data captured with the right camera 61 and/or the leftcamera 62, a characteristic quantity set in advance may be used. Thesame holds true for the case where the state detection section 164detects the position, size, color, shape, and other factors of a targeton which an action is performed, a gadget or a tool used to perform theaction, the user's body part, or any other object from image datacaptured with the right camera 61 and/or the left camera 62. Thecharacteristic quantity is data used to detect an image from a capturedimage and is a characteristic quantity or any other parameter of theimage. For example, to detect a target that is an object, the settingdata contains a characteristic quantity representing the color, shape,or any other characteristic of a captured image of the object. In thiscase, the input recognition section 162 and the state detection section164 extract an image of the object from image data on an outside sceneimage, calculate a characteristic quantity of the extracted image of theobject, and compare the calculated characteristic quantity with thecharacteristic quantity contained in the setting data 121 and verify thedegree of coincidence. When the comparison shows close similarity orcomplete coincidence, the object in the image extracted from the outsidescene image can be recognized as the target. When the setting data 121contains a plurality of characteristic quantities of a target, the inputrecognition section 162 and the state detection section 164 can detectthe target from an outside scene image on the basis of the plurality ofcharacteristic quantities and recognize the target.

How the input recognition section 162 and the state detection section164 recognize a target and an indicating tool from an image is notlimited to the method for recognizing an image of the target on thebasis of a characteristic quantity of the image as described above. Forexample, the user may indicate a target or an indicating tool, forexample, from objects contained in an outside scene image to select thetarget or the indicating tool. In this case, the indication performed bythe user may be voice indication, and the voice processing section 187converts the voice collected through the microphone 63 into a text,which allows the input recognition section 162 and the state detectionsection 164 to acquire information for recognition and identification ofthe target. For example, when voice that specifies a characteristic of atarget in a captured image, such as the color or the shape of thetarget, is converted into a text, the input recognition section 162 andthe state detection section 164 detect an image corresponding to thespecified characteristic from the captured image and recognize theimage. A method for inputting information on a target can be a methodusing an input detectable by the input recognition section 162, asdescribed above.

The evaluation section 166 evaluates the state of a user of the HMD 100.More specifically, the evaluation section 166 evaluates whether or notthe state of the user is a state in which the notification data 132should be transmitted from the HMD 100 to another HMD 100. The referencein accordance with or the condition under which it is determined thatthe notification data 132 should be transmitted is set in advance andstored, for example, in the storage section 120 as the setting data. Forexample, the reference in accordance with which the determination ismade can be a situation in which the input recognition section 162 hasdetected and recognized a predetermined input. In this case, when theinput recognition section 162 has recognized an input corresponding to acondition set in advance, the evaluation section 166 determines that theuser's state is the state in which the notification data 132 should betransmitted.

The reference or the condition used by the evaluation section 166 forthe evaluation is not limited to a content relating to an inputrecognized by the input recognition section 162. For example, theevaluation section 166 may evaluate whether or not motion of the user'shead or body corresponds to preset motion on the basis of a detectionvalue from the nine-axis sensor 66. Instead, the evaluation section 166may evaluate whether or not motion of the user's sight lines correspondsto preset motion on the basis of a detection value from the sight linesensors 68. Still instead, the HMD 100 may be provided with a vitalsensor, such as an electromyograph (not shown), a pulse measurementapparatus (not shown), a blood pressure measurement apparatus (notshown), or a blood oxygen level measurement apparatus (not shown), andthe evaluation section 166 may perform the evaluation on the basis of ameasurement value or a detection value from the vital sensor. In any ofthese cases, even when the user has no intention of transmitting thenotification data 132, the state of the user's body can be evaluated.For example, when the user desires to transmit the notification data 132but cannot issue an input recognizable by the input recognition section162, the evaluation section 166 can evaluate the user's state.

Further, the evaluation section 166 may learn an input recognized by theinput recognition section 162, detection values from the nine-axissensor 66 and the sight line sensors 68, and a detection value from thevital sensor and set or change the reference in accordance with or thecondition under which the user's state is evaluated. The learning may beautonomous learning by providing the evaluation section 166 with an AI(artificial intelligence) engine. The evaluation section 166 may insteadoptimize the evaluation reference or condition in accordance with analgorithm set by the evaluation section 166 in advance.

In the example shown in FIG. 2, when a problem occurs in the workperformed by the worker UB, it is appropriate that the worker UBtransmits the notification data 132 to the administrator UA or theworker UC, but it is conceivable that the worker UB is absorbed insolving the problem and does not operate the HMD 100. In this case,evaluation of the state of the worker UB allows not only detection ofoccurrence of a problem in the work line and identification of thelocation where the problem has occurred but also assistance or othercountermeasures taken by another worker or the administrator. Subsequenttransmission of the notification data 132 and images captured with theright camera 61 and/or the left camera 62 allows the administrator UA orthe worker UC other than the worker UB, who is the person in trouble, toquickly grasp the situation of the problem having occurred.

The communication control section 165 controls the communication section117 to cause it to perform wireless communication with a wireless accesspoint (not shown) or any other device connected to the communicationnetwork 4.

When the plurality of HMDs 100 that form the display system 1 arelocated in positions close to one another, that is, when the pluralityof HMDs 100 are located in a range over which the communication section117 of one of the HMDs 100 can transmit and receive a wireless signal toand from the communication section 117 of another HMD 100, directwireless communication between the communication sections 117 may beperformed without via the communication network 4.

FIGS. 9A and 9B are flowcharts showing the action of the display system1.

The following description will be made of the case where theadministrator UA, who administers work, wears and uses the HMD 100A, theworker UB wears and uses the HMD 100B, and the worker 15C wears and usesthe HMD 100C, as shown in FIGS. 1 and 2.

FIG. 9A shows the action of the HMD 100B, and FIG. 9B shows the actionof the HMD 100A.

To start work, an action start instruction is inputted to the HMD 100A(step S31), and the action start instruction is inputted also to the HMD100B (step S11).

The HMD 100A transmits the action guidance data 133 stored in advance inthe transmission data storage section 126 (step S32), and the HMD 100Breceives the action guidance data 133 and stores it in the receptiondata storage section 127 (step S12).

The AR display control section 163 of the FIND 100B starts, for example,displaying a still image, motion images, a text, or any other type ofinformation and/or outputting voice on the basis of the action guidancedata 133 in the reception data storage section 127 (step S13) The workerUB, who wears the HMD 100B, can therefore perform work on the target OB(FIG. 2) in accordance with guidance based on the action guidance data133.

The input recognition section 162 of the HMD 100B starts image capturingby using the right camera 61 and/or the left camera 62 and voicedetection by using the voice processing section 187 when the work starts(step S14).

The HMD 100B uses the function of the input recognition section 162 togenerate the visual field data 131 from image data captured with theright camera 61 and/or the left camera 62, stores the visual field data131 in the transmission data storage section 126, and transmits thevisual field data 131 to the HMD 100A (step S15). The cycle of thegeneration and transmission of the visual field data 131 is specified bydata contained in the action guidance data 133 or the setting data 121.

The input recognition section 162 evaluates whether or not an input hasbeen detected (step S17). When an input has been detected (step S17:YES), the input recognition section 162 generates notification data 132corresponding to the content of the input, stores the notification data132 in the transmission data storage section 126, transmits thenotification data 132 to the HMD 100A (step S18), and transitions tostep S19. When a result of the evaluation shows that no input has beendetected (step S17: NO), the input recognition section 162 transitionsto step S19.

In step S19, the state detection section 164 evaluates whether or notthe action state has changed (step S19). When the action state haschanged (step S19: YES), for example, when an action has been initiatedor when an action has been completed, the state detection section 164updates the state information 133 c in the action guidance data 133stored in the reception data storage section 127 (step S20) andtransitions to step S21. When the action state has not changed (stepS19: NO), the state detection section 164 transitions to step S21.

When the visual field data 131 has been transmitted from the HMD 100B tothe HMD 100A, the HMD 100A receives the visual field data 131 and storesit in the reception data storage section 127 (step S34). The AR displaycontrol section 163 of the HMD 100A outputs a still image, motionimages, voice, or any other type of information corresponding to thevisual field of the worker UB who wears the HMD 100B on the basis of thereceived visual field data 131 (step S35).

When the HMD 100B has transmitted the notification data 132 to the HMD100A, the HMD 100A receives the notification data 132 and stores it inthe reception data storage section 127 (step S36). The AR displaycontrol section 163 of the HMD 100A outputs a text, a still image,motion images, voice, or any other type of information representing thecontent of the input in the HMD 100B on the basis of the receivednotification data 132 (step S37).

The input recognition section 162 of the HMD 100A evaluates whether ornot an input has been detected (step S38). When an input has beendetected (step S38: YES), the input recognition section 162 generatesnotification data 132 corresponding to the content of the input, storesthe notification data 132 in the transmission data storage section 126,transmits the notification data 132 to the HMD 100B (step S39), andtransitions to step S40. When a result of the evaluation shows that noinput has been detected (step S38: NO), the input recognition section162 transitions to step S40.

In step S39, the input recognition section 162 of the HMD 100A maytransmit the notification data 132 to the plurality of HMDs 100E and100C at the same time. In this case, the same notification data 132 maybe transmitted to the plurality of HMDs 100B and 100C, or differentpieces of notification data 132 may be generated and transmitted to theplurality of HMDs 100E and 100C. The transmission destination may bespecified in accordance with an input from the administrator UA wheneverthe transmission is performed or may be determined in accordance withthe transmission destination setting data 122.

In step S23, the control section 140 of the HMD 100E evaluates whetheror not an end condition has been satisfied (step S23). The end conditionis satisfied, for example, when all actions contained in the actionguidance data 133 received in step S12 have been completed or when theinput recognition section 162 has detected an end instruction input.When the end condition has not been satisfied (step S23: NO), thecontrol section 140 returns to step S15. When the end condition has beensatisfied (step S23: YES), the control section 140 terminates thepresent procedure. At the time of termination of the procedure, thecontrol section 140 may duplicate the action guidance data 133 stored inthe reception data storage section 127, store the duplicated actionguidance data 133 in the transmission data storage section 126, andtransmit the duplicated action guidance data 133 to the HMD 100A.Further, the control section 140 may transmit notification data 132 thatnotifies the HMD 100A of the action termination.

In step S40, the control section 140 of the HMD 100A evaluates whetheror not an end condition has been satisfied (step S40). The end conditionis satisfied, for example, when the HMD 100B has notified the HMD 100Aof action termination or when the input recognition section 162 hasdetected an end instruction input. When the end condition has not beensatisfied (step S40: NO), the control section 140 returns to step S34.When the end condition has been satisfied (step S40: YES), the controlsection 140 terminates the present procedure. At the time of terminationof the procedure, the control section 140 may transmit notification data132 that notifies the termination to the other HMDs 100 contained in thedisplay system 1. In this case, an HMD 100 other than the HMD 100A usedby the administrator UA may be set to terminate any action when the HMD100 receives the notification data 132 that notifies the terminationfrom the HMD 100A.

In the flowchart shown in FIG. 9A, the HMD 100B carries out the processof detecting an input (step S17), the process of detecting a change inthe action state (step S19), the process of receiving the notificationdata 132 (step S21), and the process of evaluating the termination (stepS23) sequentially on a step basis, but the invention is not necessarilyconfigured this way. The action of each of the steps described above maybe performed as an interruption process. That is, the control section140 of the HMD 100B starts operating in step S11 and is then ready forthe input detection (step S17), the detection of a change in the actionstate (step S19), the reception of the notification data 132 (step S21),and the instruction of the termination (step S23). When an input or anaction is detected, the action of the corresponding one of the stepsS17, S19, S21, and S23 may be initiated as an interruption process. Thesame holds true for the steps S34, S38, S39, and S40 in FIG. 9B.

FIGS. 10A and 10B are flowcharts showing the action of the displaysystem 1 and show an action in which data on actions are taken overbetween the HMDs 100E and 100C used by the workers UB and UC who performthe actions. FIG. 10A shows the action of the HMD 100B, and FIG. 10Bshows the action of the HMD 100C. FIGS. 10A and 10B show a case where anaction is taken over from the HMD 100B to the HMD 100C.

The action in FIGS. 10A and 10B is performed during a period for whichthe HMD 100B performs an action in accordance with the action guidancedata 133, that is, during a period between steps S13 to S23 in FIG. 9A.

The input recognition section 162 of the HMD 100B detects an input of anaction takeover instruction (step 951) and then detects an input of anaction takeover request to an HMD 100 that takes over the action (stepS52). The input recognition section 162 generates notification data 132on the basis of the content of the input relating to the action takeoverrequest, stores the notification data 132 in the transmission datastorage section 126, and transmits the notification data 132 to the HMD100C (step 953).

The AR display control section 163 of the HMD 100C receives thenotification data 132 from the HMD 100B, stores the notification data132 in the reception data storage section 127 (step S61), and outputs atext, a still image, motion images, voice, or any other type ofinformation on the basis of the received notification data 132 (stepS62).

The control section 140 of the HMD 100B duplicates the action guidancedata 133 stored in the reception data storage section 127, stores theduplicated action guidance data 133 in the transmission data storagesection 126, and transmits the duplicated action guidance data 133 tothe HMD 100C (step S54). The action guidance data 133 is data havingbeen stored in the reception data storage section 127 and updated inaccordance with the progress of the action in the HMD 100B.

The control section 140 of the HMD 100C receives the action guidancedata 133 and stores it in the reception data storage section 127 (stepS63). The AR display control section 163 produces a notification of thereception of the action guidance data 133, transmits the notification tothe HMD 100B (step S64), and starts, for example, displaying a stillimage, motion images, a text, or any other type of information and/oroutputting voice on the basis of the received action guidance data 133(step S65). At this point, the AR display control section 163 outputs,in accordance with the state information 133 c contained in the receivedaction guidance data 133, the guidance information 133 a formed of anaction under execution or an action not having been performed. Theworker UC, who wears the HMD 100C, can therefore follow the actionhaving been performed by the worker UB, who wears the HMD 100B, andperform an action on the target OB (FIG. 2) in accordance with theguidance based on the action guidance data 133.

The control section 140 of the HMD 100B receives the notificationrelating to the reception of the action guidance data 133 from the HMD100C (step S55) and terminates the present procedure. At this point, thecontrol section 140 of the HMD 100B may delete the action guidance data133 stored in the reception data storage section 127. The controlsection 140 of the HMD 100E may also delete the action guidance data 133stored in the transmission data storage section 126. Since the actionguidance data 133 is therefore not redundantly held by a plurality ofHMDs 100, a situation in which actions are mixed up can be avoided.

The input of the action takeover instruction and the action takeoverrequest in steps S11 and S12 may be performed in the HMD 100A. In thiscase, notification data 132 on the action takeover instruction istransmitted from the HMD 100A to the HMDs 100B and 100C, and the HMDs100B and 100C may start the action in FIGS. 10A and 103 in accordancewith the notification data 132.

FIGS. 11A to 11C show examples of display in the display system. FIG.11A shows an example in which the HMDs 100B and 100C perform display forthe workers UB and UC, and FIG. 11B shows another example in which theHMDs 100B and 100C perform display for the workers UB and UC. FIG. 110shows an example in which the HMD 100A performs display.

In FIGS. 10A to 100, reference character VR denotes the field of view ofa user (administrator UA, workers UB and UC), and reference character V1denotes a region in which an HMD 100 displays an image and allows theuser to view the image, that is, the display region of the image displaysection 20. The display region V1 is, for example, located roughly atthe center of the user's field of view VR and narrower than the field ofview VR. The size of the display region V1 may instead be equal to thesize of the field of view VR, and the size and the position of thedisplay region V1 are not limited to those in the examples shown inFIGS. 11A to 11C.

In FIG. 11A, the AR display control section 163 displays a list SR,which lists the contents of the guidance information 133 a in the actionguidance data 133. The list SR is a list in which the contents of theguidance information 133 a contained in the action guidance data 133 arearranged in accordance with the order specified by the order information133 b. The list SR may display the titles and contents of the ofguidance information 133 a. Instead, the order specified by the orderinformation 133 b may be displayed.

In the list SR, check boxes CH are disposed in correspondence with thepositions where the contents of guidance information 133 a aredisplayed. Each of the check boxes CH, when it is checked, indicatesthat the corresponding action in the action guidance data 133 has beencompleted. When the state detection section 164 detects that an actionhas been completed, the AR display control section 163 changes thedisplay state of the corresponding check box CH to a state showingaction complete. In response to the change, the state detection section164 may update the state information 133 c in the action guidance data133 in the reception data storage section 127. When the number ofcontents of guidance information 133 a contained in the action guidancedata 133 is one, the list SR contains the content of the one action andthe check box CH associated therewith.

The list SR can instead have a configuration with no check box CH. Inthis case, the AR display control section 163 may delete, from the listSR, an item corresponding to an action detected by the state detectionsection 164 that the action has been completed. Instead, the color orbrightness displayed in the list SR may be changed.

The example shown in FIG. 11B is a case where the contents of theguidance information 133 a in the action guidance data 133 is displayedone by one or on a preset number basis. In this example, the contents ofthe guidance information 133 a are displayed one by one in the displayregion V1 and in a balloon-shaped display section D. The edge of thedisplay section D may be drawn in the form of an arrow that points anaction target position of the target OB. Instead, a marker M may bedisplayed in a position on which an action corresponding to the guidanceinformation 133 a displayed in the display section D is performed. Inthe example shown in FIG. 11B, when the state detection section 164detects that an action corresponding to the guidance information 133 adisplayed in the display section D is completed, the AR display controlsection 163 displays the following guidance information 133 a containedin the action guidance data 133 in the display section D. The itemizedactions in the guidance information 133 a are thus sequentiallydisplayed in the display section D in accordance with the order inaccordance with which the actions are performed.

The display position and display form of each of the list SR and thedisplay section D can be changed as appropriate. For example, each ofthe list SR and the display section D may be displayed in a positionwhere the list SR or the display section D overlaps with the target OBon which an action corresponding to an item in the action guidance data133 is performed. However, displaying the list SR or the display sectionDin a position where the list SR or the display section D does notoverlap with the target OB is advantageous in that the list SR or thedisplay section D does not prevent the action and allows the action tobe readily performed while the item is viewed. Instead, the list SR orthe display section D may be displayed in a 3D (stereoscopic) form or asa 2D (planar) image. Still instead, the list SR or the display section Dmay be displayed only by one of the right display driver 22 and the leftdisplay driver 24.

In FIGS. 11A and 11B, a pointer P, which indicates the direction of theuser's sight lines detected with the sight line sensors 68, may bedisplayed in the display region V1.

The example shown in FIG. 11C is an example in which the HMD 100Adisplays still images based on the visual field data 131 transmittedfrom the HMDs 100B and 100C. In the display region V1, a visual fielddisplay VW1 based on the visual field data 131 from the HMD 100B and avisual field display VW2 based on the visual field data 131 from the HMD100C are displayed side by side. In the visual field displays VW1 andVW2, IDs representing the HMDs 100B and 100C, which are transmissionsources, are displayed in correspondence therewith. In this case, theadministrator UA can view the visual fields of the workers UB and UC atthe same time.

Further, in the visual field displays VW1 and VW2 are displayed thepointer P indicating the direction of the sight lines of the worker UBand the pointer P indicating the direction of the sight lines of theworker UC, respectively. The pointer P can be displayed when the visualfield data 131 contains data representing the position where the pointerP is displayed.

As described above, in the display system 1, among the plurality of HMDs100, the HMD 100A, which has been set in advance as the HMD 100 for anadministrator, can display the visual fields of the other HMDs 100E and100C in the form of images. Further, the content of an input or acontent corresponding to the input in each of the HMDs 100E and 100C istransmitted to the HMD 100A in the form of the notification data 132,and the HMD 100A can output the notification data 132. Therefore, in thedisplay system 1, the action guidance data 133 allows assistance ofactions performed by a plurality of performers who perform the actionsand grasp of situations in which the actions are performed.

The input to the HMD 100 can be performed in the form of voice, asdescribed above, and a gesture can be detected for GUI operation of theHMD 100. Further, input through a foot switch and knock operationperformed on the image display section 20 are also allowed. Inputcontents corresponding to the input to a foot switch and the input bythe knock operation may be set in advance in the input recognitionsection 162. For example, when the foot switch is turned on, a messageset in advance by the setting data 121 may be transmitted as thenotification data 132. Further, for example, knock operation performedon the image display section 20 may trigger transmission of image datacaptured with the right camera 61 and the left camera 62 to the HMD 100Aas the notification data 132 along with a message or after preset imageprocessing is performed on the image data. In this case, the footswitch, the knock operation, or any other input operation allowsemergency contact with the administrator UA, a request for assistance,or any other action. Further, the transmission destination in the casewhere input through the foot switch, input of the knock operation, orany other type of input is detected may be set by the transmissiondestination setting data 122. In this case, for example, in the event ofan emergency, an emergency message can be transmitted from one HMD 100to the other HMDs 100 for display of the emergency message.

The assistance and administration of the workers (performers) using theplurality of HMDs 100 can be administered by the HMD 100A in theadministrator's site A provided in a separate room or at a remotelocation, whereby the point at which a performer's gaze is directed andthe content of an action of the performer can be checked. The work inthe work line FA can be quickly checked, for example, by measurement oflost time for each performer who wears an HMD 100 and discovery ofoccurrence of a defect due to work failure. In this case, lost time inthe work line FA can be shortened, and the overall throughput can beimproved.

Further, causing the plurality of HMDs 100 to provide output based onthe action guidance data 133 allows work assistance images to bedisplayed as teaching images for unification of work order andsynchronization of workers with one another. When a performer needs tobe replaced, work can be taken over between HMDs 100, as described withreference to FIGS. 10A and 10B, whereby the same work order can be takenover.

Further, making use of the display system 1 allows evaluation of whetheror not the worker who uses any of the HMDs 100 among the HMDs 100connected to one another via the communication network 4 is idle. Theadministrator UA can therefore issue information that allows efficientsupply of a necessary member to an idle worker's site.

Further, the function of transmitting the notification data 132 from theHMD 100A to each of the HMDs 100E and 100C and causing the HMD 100 tooutput the notification can be used to cause the image display section20 to display an indicator as a sign that prompts supply of a necessarymember. Exchange of information among the wearers of the HMDs 100 asdescribed above allows smooth supply of parts, work takeover, and othertypes of operation, whereby a series of work can be continuouslymaintained.

FIGS. 12A to 12D are descriptive diagrams showing a specific applicationexample of the invention. In the specific application example, aplurality of workers UB, UC, and UD wear HMDs 100, an administrator UAwears an HMD 100, and the workers UB, UC, and UD work under theadministration of the administrator UA, as in the work line shown inFIG. 2. The HMD 100 worn by the administrator UA is called an HMD 100A,and the HMDs 100 worn by the workers UB, UC, and UD are called HMDs100B, 100C, and 100D, respectively. Each of the HMDs 100A, 100B, 100C,and 100D has the configuration of the HMD 100 described above.

When a trouble has occurred in the work performed by the worker UB, andthe evaluation section 166 of the HMD 100B has determined that the stateof the worker UB is the state in which notification data 132 should betransmitted, the notification data 132 is transmitted from the HMD 100Bto the HMD 100A (step ST1), as shown in FIG. 12A. The destination towhich the notification data 132 is transmitted is set by thetransmission destination setting data 122 described above and is, forexample, a group including the HMDs 100A, 100C, and 100D. Thenotification data 132 can notify the administrator UA and the workers UCand UD of the occurrence of a problem with the worker UB.

Subsequently, the administrator UA operates the HMD 100A to cause it totransmit notification data 132 containing an instruction to the HMD 100C(step ST2), as shown in FIG. 12B. Instead, the worker UC operates theHMD 100C to cause it to transmit notification data 132 on assistanceoffer to the HMD 100A (step ST3). The mutual transmission and receptionof the notification data 132 allow determination of replacement of theworker UB with the worker UC.

Thereafter, the administrator UA operates the HMD 100A to cause it totransmit notification data 132 representing that the worker UC replacesthe worker UB to the HMDs 100B and 100D (step ST4), as shown in FIG.12C.

The worker UB then leaves the work line, and the worker UC moves to thework site in place of the worker UB. At this point, the worker UCprovides an input to the HND 100C to cause it to transmit notificationdata 132 representing completion of the replacement to the HMDs 100A and100D.

When the worker UB leaves, action guidance data may be transmitted fromthe END 100B to the HMD 100C in a period between the state in FIG. 12Cand the state in FIG. 12D.

As described above, application of the invention allows quick discoveryof time lost by a single user in in-line work and occurrence of a defectdue to work failure, whereby lost time in the entire in-line work can beshortened, and the overall throughput can be improved. Each worker canreadily and quickly share and check the work of the worker and action ofconduct of the worker with another HMD wearer for improvement inreliability of team work and conduct. Further, the users who wear theHMDs 100 can share information, and an HMD 100 can share informationwith the other HMDs 100 in parallel to one another, in a loop form, orin series. An administrator is not necessarily stationed in a vicinityof a work line and may be stationed in any position where theadministrator can administer the work line, and the administrator s HMD100 can further monopolistically collect and administer information. Forexample, setting the transmission destination setting data 122 and thereference or condition of evaluation made by the evaluation section 166allow proper use of the range over which the information is shared andthe procedure in accordance with which information is shared dependingon TPO (use application).

As described above, the display system 1 according to the embodiment towhich the invention is applied includes a plurality of HMDs 100, whichare head-mounted-type display apparatus. Each of the HMDs 100 includesthe image display section 20, which allows a user to view an image andtransmits an outside scene, and the communication section 117, whichcommunicates with the other HMDs 100. Each of the HMDs 100 furtherincludes the right camera 61 and the left camera 62, which perform imagecapturing over a range that overlaps with the user's visual field, theevaluation section 166, which evaluates the user's state, and thecontrol section 140. The control section 140 transmits the visual fielddata 131 on the basis of the captured images to another HMD 100 andtransmits the notification data 132 in accordance with a result of theevaluation performed by the evaluation section 166 to another HMD 100via the communication section 117. The control section 140 furthercauses the image display section 20 to perform display on the basis ofdata transmitted from another HMD 100. In the display system 1, data onthe visual fields and data on inputs can therefore be shared among theplurality of HMDs 100. As a result, a plurality of users who wear theHMDs 100 can share the visual fields and operation. For example, aperson who performs an action and a person who assists the action wearthe HMDs 100, which are head-mounted-type display apparatus, forconveyance of a large amount of information on actions in a moreunderstandable manner. A user can therefore assist another user whoperforms work or any other type of act or readily and accuratelyadminister the state in which work or any other type of act isperformed.

Since the input recognition section 162 recognizes an input on the basisof voice detected by the voice processing section 187, the HMD 100 canaccept an input in the form of voice, whereby information can be morereadily conveyed among a plurality of HMDs 100.

The control section 140 transmits data representing an input recognizedby the input recognition section 162 on the basis of voice detected bythe voice processing section 187 to another END 100. A content inputtedin the form of voice to an HMD 100 can therefore be notified to a userwho wears another HMD 100.

The HMD 100 includes the input recognition section 162, which recognizesan input, and the evaluation section 166 evaluates the state of the useron the basis of an input recognized by the input recognition section162. The HMD 100 can therefore recognize an input from the user oranother person to appropriately evaluate the state of the user. Further,the user or another person allows the HMD 100 to recognize an input tointentionally cause the HMD 100 to transmit notification data to anotherapparatus.

The HMD 100 further includes the nine-axis sensor 66, which detectsmotion of the image display section 20, and the input recognitionsection 162 may instead recognize an input on the basis of motiondetected with the nine-axis sensor 66. In this case, the inputrecognition section 162 can accept an input according to motion of theimage display section 20 or motion of the user who wears the imagedisplay section 20.

The input recognition section 162 may instead recognize an input on thebasis of images captured with the right camera 61 and the left camera62. In this case, the input recognition section 162 can more readilyaccept an input.

The control section 140 carries out a reproduction process ofreproducing the action guidance data 133 including the guidanceinformation 133 a, which gives guidance on actions, the orderinformation 133 b, which specifies the order in accordance with whichthe actions are performed, and the state information 133 c, which showsthe state of the performed actions. In the reproduction process, thecontrol section 140 causes the image display section 20 to display atext, a still image, motion images, or any other type of information.The control section 140 updates, on the basis of an input recognized bythe input recognition section 162, the state information 133 c in theaction guidance data 133 being reproduced and causes the communicationsection 117 to transmit the action guidance data 133 having undergonethe reproduction process to another HMD 100. As a result, actionguidance can be given to a user who wears an HMD 100 and performs anaction, and data containing records on the action performed inaccordance with the guidance can be taken over and used by the other HMD100.

The invention is not limited to the configuration of the embodimentdescribed above and can be implemented in a variety of aspects to theextent that they do not depart from the substance of the invention.

In the embodiment described above, the configuration in which a userviews an outside scene that passes through the display section is notlimited to the configuration in which the right optical image displaysection 26 and the left optical image display section 28 transmitoutside light. For example, the invention is also applicable to adisplay apparatus that displays an image but does not allow a user toview an outside scene. Specifically, the invention is applicable to adisplay apparatus that displays images captured with the right camera 61and/or the left camera 62, an image and a CG produced on the basis ofthe captured images, video images based on prestored video data orexternally inputted video data, and other types of image. An example ofa display apparatus of this type may include a display apparatus thatdoes not allow a user to view an outside scene or what is called aclosed-type display apparatus. Further, a display apparatus that doesnot perform AR display, MR display, or VR display but displaysexternally inputted video data or an analog video signal is, of course,an apparatus to which the invention is applied.

Further, for example, the image display section 20 may be replaced withan image display section worn, for example, as a cap or any other imagedisplay section worn based on another method. A display section thatdisplays an image in correspondence with a user's left eye and a displaysection that displays an image in correspondence with the user's righteye only need to be provided. Moreover, the display apparatus accordingto the embodiment of the invention may, for example, be configured as ahead mounted display incorporated in an automobile, an airplane, andother vehicles. Further, for example, the display apparatus may beconfigured as a head mounted display built in a helmet or other bodyprotection gears. In this case, a portion that determines the positionrelative to a user's body and a portion that is positioned relative tothe positioning portion can be a portion worn by the user.

Further, in the embodiment described above, the description has beenmade of the case where the image display section 20 is separated fromthe control device 10 and they are connected to each other via theconnection section 40. The control device 10 and the image displaysection 20 can instead be integrated with each other, and the integratedunit can be worn around a user's head.

The control device 10 may be a notebook computer, a tablet computer, ora desktop computer. Instead, the control device 10 may, for example, bea mobile electronic apparatus including a game console, a mobile phone,a smartphone, and a mobile media player, or any other dedicatedapparatus. Further, the control device 10 may be configured to beseparate from the image display section 20, and a variety of signals maybe transmitted and received between the control device 10 and the imagedisplay section 20 over wireless communication.

Further, for example, the configuration that generates image light inthe image display section 20 may be a configuration including an organicEL (organic electro-luminescence) display and an organic EL controlsection. Moreover, an LCOS (liquid crystal on silicon) device (LCoS is aregistered trademark), a digital micromirror device, or any other devicecan be used as the configuration that generated image light.

The “display section” used herein corresponds to a configuration thatoutputs image light, and output of image light from the HMD 100 iscalled “display” in the following description.

The embodiment described above illustrates the configuration in whichthe right and left image light generation units described with referenceto FIGS. 5A and 5B generate image light and the right optical imagedisplay section 26 and the left optical image display section 28 shownin FIG. 3 radiate the image light toward the user's right and left eyesto cause the image light to be incident on the user's right and lefteyes. The configuration of the “display section” is not limited to theconfiguration described above. That is, any configuration that radiatesthe image light other than the configuration shown in FIG. 3 and FIGS.5A and 5B can be used. For example, in the configuration of the presentembodiment, the “right light guide unit” and the “left light guide unit”having the half-silvered mirrors 261A and 262A output the image lighttoward the user's eyes. As the configuration that generates image light,the right backlight 221 and the left backlight 222 as well as the rightLCD 241 and the left LCD 242 are provided. The “display section” doesnot require this configuration as an essential portion.

For example, image light generated by a mechanism in which one or bothof the right display driver 22 and the left display driver 24 of theimage display section 20 are built may be reflected off a reflectionmechanism provided on the user's side of the image display section 20,that is, the side facing the user's eyes and outputted toward the user'seyes. The reflection mechanism can, for example, be a sweep system usinga MEMS (micro electro mechanical systems) mirror. That is, aconfiguration in which a sweep system having a MEMS mirror that sweepsthe light outputted from the image light generation units is providedand the light swept by the sweep system is directly incident on theuser's eyes may be employed. Further, the image display section 20 maybe provided with an optical member on which a virtual image is formed bythe light swept by the sweep system. The optical member uses the lightswept with the MEMS mirror to form a virtual image. In this case, whenthe MEMS mirror sweeps light, a virtual image is formed on a virtualimage formation plane, and the user captures the virtual image with theeyes to view (recognize) the image. The optical part in this case may bea part that guides light by reflecting the light multiple times, forexample, the right light guide plate 261 and the left light guide plate262 in the embodiment described above, or may be a half-silveredsurface.

The sweep system is not limited to the configuration including a MEMSmirror. The mechanism that generates image light may also instead be alaser light source that emits laser light. For example, the invention isalso applicable to a laser-retina-projection-type head mounted display.That is, a configuration in which a light output section may include alaser light source and an optical system that guides the laser beam fromthe laser light source to user's eyes may be employed. In thisconfiguration, the laser beam is caused to be incident on each of theuser's eyes, and the laser beam is swept over the retina to form animage on the retina, so that the user is allowed to view the image.

Instead, in place of the virtual formation plane that receives the sweptlight, a diffraction grating may be used to guide the image light to theuser's eyes. That is, the configuration in which the image light isguided through an optical member is not necessarily employed, and aconfiguration having only a function of guiding the image light towardthe user's eyes by refracting and/or reflecting the image light.

In the configuration in which a sweep system having a MEMS or any othercomponent, changing the angle at which the sweep system is attached tothe image display section 20 allows the position where the user views animage, that is, the position where an image is displayed to be changed.Therefore, in the process of changing the image display position in theembodiment described above, the angle of the sweep system may be changedinstead of changing the position where images are displayed in the rightLCD 241 and the left LCD 242.

As the optical system that guides image light to the user's eyes, anemployable configuration includes an optical member that transmitsexternal light externally incident on the display apparatus and allowsthe external light along with image light to be incident on the user'seyes. Another usable optical system may be an optical member that isdisposed in a position in front of the user's eyes and overlaps withpart of the visual field of the user or coincides with the entire visualfield of the user.

In the embodiment described above, the configuration in which thehalf-silvered mirrors 261A and 262A form virtual images on part of theright optical image display section 26 and the left optical imagedisplay section 28, which are located in front of the user's eyes isillustrated. The invention is not limited to the configuration describedabove, and a configuration in which an image is displayed in a displayregion that occupies the entire or majority of the right optical imagedisplay section 26 and the left optical image display section 28 may beemployed. In this case, the process of reducing the size of an image maybe included in the action of changing the position where the image isdisplayed.

Further, the optical element in the invention are not limited to theright light guide plate 261 and the left light guide plate 262 havingthe half-silvered mirrors 261A and 262A and only needs to be an opticalpart that causes image light to be incident on the user's eyes.Specifically, a diffraction grating, a prism, or a holography displaysection may be used.

At least part of the functional blocks shown in FIG. 6 may be achievedby hardware or hardware and software cooperating with each other, andthe configuration formed of independent hardware resources shown in FIG.6 is not necessarily employed. The program executed by the controlsection 140 may be stored in the storage section 120 or a storage devicein the control device 10, or a program stored in an external device maybe acquired via the communication section 117 or the interface 125 andexecuted. Among the configurations formed in the control device 10, onlythe operation section 135 may be formed as an independent user interface(UI). Further, the configurations formed in the control device 10 may beredundantly formed in the image display section 20. For example, thecontrol section 140 shown in FIG. 6 may be formed both in the controldevice 10 and the image display section 20, and the control section 140formed in the control device 10 and the CPU formed in the image displaysection 20 may perform different functions.

The entire disclosure of Japanese Patent Application No. 2015-135194,filed Jul. 6, 2015 is expressly incorporated by reference herein.

What is claimed is:
 1. A system comprising a plurality ofhead-mounted-type display apparatus, wherein each of the displayapparatus includes a display section that allows a user to view an imageand transmits an outside scene, a communication section thatcommunicates with another display apparatus of the plurality of displayapparatus, an imaging section that performs image capturing over a rangethat overlaps with a visual field of the user, an evaluation sectionthat evaluates a state of the user, and a control section, wherein thecontrol section transmits visual field data on the basis of an imagecaptured by the imaging section to the another display apparatus, causesthe communication section to transmit notification data on the basis ofa result of the evaluation performed by the evaluation section to theanother display apparatus, and causes the display section to performdisplay on the basis of data transmitted from the another displayapparatus.
 2. The display system according to claim 1, wherein theevaluation section evaluates, on the basis of an action of the user,whether the state of the user corresponds to a state in whichnotification set in advance should be made, and when the evaluationsection determines that the state of the user corresponds to the statein which notification should be made, the control section causes thecommunication section to transmit notification data to the anotherdisplay apparatus.
 3. The display system according to claim 1, furthercomprising a recognition section that recognizes an input, wherein theevaluation section evaluates the state of the user on the basis of theinput recognized by the recognition section.
 4. The display systemaccording to claim 3, further comprising a voice processing section thatdetects voice, wherein the recognition section recognizes an input onthe basis of the voice detected by the voice processing section.
 5. Thedisplay system according to claim 4, wherein the control sectiontransmits data representing the input recognized by the recognitionsection on the basis of the voice detected by the voice processingsection to the another display apparatus.
 6. The display systemaccording to claim 3, further comprising a motion detection section thatdetects motion of the display section, wherein the recognition sectionrecognizes an input on the basis of the motion detected by the motiondetection section.
 7. The display system according to claim 3, whereinthe recognition section recognizes an input on the basis of an imagecaptured by the imaging section.
 8. The display system according toclaim 3, wherein the control section performs a reproduction process ofreproducing action guidance data containing guidance information thatgives guidance on actions, order information that specifies an order inaccordance with which the actions are performed, and state informationthat shows a state of the performed actions, causes the display sectionto perform display in the reproduction process and updates, on the basisof an input recognized by the recognition section, the state informationin the action guidance data being reproduced, and causes thecommunication section to transmit the action guidance data havingundergone the reproduction process to the another display apparatus. 9.The display system according to claim 1, wherein the imaging sectionperforms image capturing over a range that overlaps with a direction inwhich the user gazes.
 10. Ahead-mounted-type display apparatuscomprising: a display section that allows a user to view an image andtransmits an outside scene; a communication section that communicateswith another head-mounted-type display apparatus; an imaging sectionthat performs image capturing over a range that overlaps with a visualfield of the user; an evaluation section that evaluates a state of theuser; and a control section, wherein the control section transmitsvisual field data on the basis of an image captured by the imagingsection to the another display apparatus, causes the communicationsection to transmit notification data on the basis of a result of theevaluation performed by the evaluation section to the another displayapparatus, and causes the display section to perform display on thebasis of data transmitted from the another display apparatus.
 11. Amethod for controlling a head-mounted-type display apparatus including adisplay section that allows a user to view an image and transmits anoutside scene, the method comprising: performing image capturing over arange that overlaps with a visual field of the user; evaluating a stateof the user; transmitting visual field data on the basis of capturedimage data to another head-mounted-type display apparatus; transmittingnotification data on the basis of a result of the evaluation of thestate of the user to the another display apparatus; and causing thedisplay section to perform display on the basis of data transmitted fromthe another display apparatus.
 12. A program executable by a computerthat controls a head-mounted-type display apparatus including a displaysection that allows a user to view an image and transmits an outsidescene, the program causing the computer to perform image capturing overa range that overlaps with a visual field of the user; evaluate a stateof the user; transmit visual field data on the basis of captured imagedata to another head-mounted-type display apparatus and transmitnotification data on the basis of a result of the evaluation of thestate of the user to the another display apparatus; and cause thedisplay section to perform display on the basis of data transmitted fromthe another display apparatus.